<html>
<head><meta charset="utf-8"><title>str-from-utf8-with-validation · wg-secure-code · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/index.html">wg-secure-code</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html">str-from-utf8-with-validation</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="174013854"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174013854" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174013854">(Aug 24 2019 at 00:14)</a>:</h4>
<p>So here's an interesting challenge: I'm performing some validation on some characters that are in a <code>&amp;[u8]</code>. Then I'm creating an <code>&amp;str</code> from it. <br>
Right now my choices are to either<br>
a) Do 2x passes over the data, once for my validation, and once in <code>str::from_utf8</code> to verify they really are utf8 (even though my validation also ensures they are utf8,<br>
b) Use the unsafe <code>str::from_utf8_unchecked</code> API to just skip the second validation</p>
<p>Neither of these appeals to me. What API _should_  exist to allow me to do everything in one pass, safely.</p>
<p>If there was <code>from_utf8_extra_check(&amp;[u8], fn(char) -&gt; Result&lt;(), Error&gt;) -&gt; Result&lt;&amp;str, Error&gt;</code> I think that'd work?</p>



<a name="174016820"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174016820" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Nick12 <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174016820">(Aug 24 2019 at 01:28)</a>:</h4>
<p>That would also be unsafe, i think</p>



<a name="174016826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174016826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Nick12 <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174016826">(Aug 24 2019 at 01:28)</a>:</h4>
<p>If the fn didnt correctly validate itd be ub in safe code</p>



<a name="174016827"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174016827" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Nick12 <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174016827">(Aug 24 2019 at 01:28)</a>:</h4>
<p>You could make it safe by having an unsafe trait</p>



<a name="174016833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174016833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Nick12 <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174016833">(Aug 24 2019 at 01:29)</a>:</h4>
<p>But youll still need unsafe to impl your validator</p>



<a name="174016835"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174016835" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Nick12 <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174016835">(Aug 24 2019 at 01:29)</a>:</h4>
<p>Whats wrong with option b?</p>



<a name="174016887"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174016887" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Nick12 <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174016887">(Aug 24 2019 at 01:30)</a>:</h4>
<p>Oh wait, the utf validation would happen outside your fn? So you validate only additional things? Im not sure i see how thats different from option a</p>



<a name="174016905"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174016905" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174016905">(Aug 24 2019 at 01:31)</a>:</h4>
<p>You'd do <code>for b in data { user_verify(b)?; normal_utf8_verify(b)?; }</code> and hope the optimizer sorted it out</p>



<a name="174028225"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174028225" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Hanna Kruppe <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174028225">(Aug 24 2019 at 07:31)</a>:</h4>
<p>I don't expect that this would be a clear performance win. Sure, you're only going through the entire string start to finish once, but<br>
1. lots of strings are short enough it doesn't matter -- if your string fits into the cache (many KB for L1, and L2/L3 might be fine too) walking it twice back-to-back is likely free, and prefetching might make it irrelevant anyway<br>
2. utf-8 validation can be optimized a lot by validating more than one char at a time and doing less work than full decoding into <code>char</code> -- having to add in actual <em>decoding</em> and doing the other validation char-by-char will not place nicely with that, likely throttling the utf-8 validation even on an algorithmic level</p>



<a name="174049497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174049497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174049497">(Aug 24 2019 at 18:27)</a>:</h4>
<p>IIRC str::from_utf8 tries pretty hard to avoid looking at characters individually, so with the extra check it would be quite a bit slower (edit: ah, whoops, someone beat me to saying this)</p>



<a name="174096235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174096235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174096235">(Aug 25 2019 at 19:08)</a>:</h4>
<p>what is the invariant on the characters in the <code>&amp;[u8]</code> for your particular use case, <span class="user-mention" data-user-id="130046">@Alex Gaynor</span> ? ASCII? something more than that?</p>



<a name="174096242"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174096242" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174096242">(Aug 25 2019 at 19:09)</a>:</h4>
<p>It's ASN.1's PrintableString that had me thinking about this, so it's an allow-list of particular ASCII chars</p>



<a name="174096244"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation/near/174096244" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tony Arcieri <a href="https://rust-lang.github.io/zulip_archive/stream/146229-wg-secure-code/topic/str-from-utf8-with-validation.html#174096244">(Aug 25 2019 at 19:09)</a>:</h4>
<p>aaaaah</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>